DNV Data Correcting and Validation
We noticed that a few critical fields, namely Pipe Length, Pipe Size, and Pipe Age, had several entries with bad data. Bad data includes negative, NULL, or blank values.
Pipe Length
There were 24 pipes with invalid lengths, either NULL or 0. The data correction process for these is a little more complicated than the other corrections and so will have it's on section following.
Correction
Length value corrections were performed in QGIS:
- Open the .shp file in QGIS.
- Filter the features where ASB_LENGTH is <= zero, NULL, or blank.
- Open the Field Calculator
- Click "Update existing field" and select ASB_LENGTH.
- In the expression box, define "$length" which equates to the geometry's length.
- Apply the changes and save the new shape file.
Validation
To validate these geometry-based values, several random pipes were selected to check. The pipe's shape was viewed in QGIS and compared to other known pipes. The new geometry-based values appear to be valid based on this comparison. Please see the several screenshots below that show a sampling of pipes along with their shapes and updated lengths.
Pipe Size/Diameter
There was only a single pipe that was missing a valid size. Pipe with ASSET_ID "SANMN07815" had a size of -99.
To fix this value, we found all adjacent pipes (of which there were three) and took their average size. Each of the adjacent pipes had a size of 200, so we changed the bad pipe to also have size 200.
Pipe Age
There were 11 pipes with AM_DATE values of -99. To fix these values, we assigned them each the oldest date in the dataset, 1943. A full list of these 11 pipes follows:
SANMN05601, SANMN05602, SANMN05438, SANMN05439, SANMN05454, SANMN05455, SANMN05779, SANMN05780, SANMN05970, SANMN05972, SANMN05973
It's also worth noting that all of the Date -99 pipes are grouped together and completely disconnected from the rest of the network. See the following screenshot:
